Repository navigation
Conversation
|
Thank you @webard, I tested locally main vs your pr branch and your pr branch looks faster 👍 main branch: 14.645 seconds. poc/parallel-lpt-buckets branch with use of |
|
Thank you @samsonasik for testing this. I'm also seeing a big improvement in my projects. However, in the larger ones the numbers show that increasing jobSize in the original implementation sometimes still gives better results. I think keeping this flag as an optional setting would be a good idea. Let's also wait for @TomasVotruba opinion. |
03d027f to
e652152
Compare
|
@webard I found that Cluster large files into one worker can improve performance on and it shows faster 1.6x from 0.64s to 0.39s: Before: structarmed-speed-01811.mp4After structarmed-0193-faster.movthat's can be next experiment for it. |
samsonasik
left a comment
samsonasik
left a comment
There was a problem hiding this comment.
Looking good to me for first experimental, additional LPT improvement can be improved more later 👍
|
@samsonasik Can you test this on laravel (https://github.com/laravel/framework) and Symfony projects ()https://github.com/symfony/symfony) and compare time+memory before/after? I want to see something heavier |
|
@TomasVotruba sure, here the result: On Laravel framework: Before : 1 minutes 30 seconds ➜ time bin/rector --no-diffs --dry-run --clear-cache --no-progress-bar ../laravel-framework/src ../laravel-framework/tests
[OK] 2683 files would have been changed (dry-run) by Rector
bin/rector --no-diffs --dry-run --clear-cache --no-progress-bar 713.50s user 16.98s system 806% cpu 1:30.56 totalAfter (with ➜ time bin/rector --no-diffs --dry-run --clear-cache --no-progress-bar ../laravel-framework/src ../laravel-framework/tests --lpt
[OK] 2683 files would have been changed (dry-run) by Rector
bin/rector --no-diffs --dry-run --clear-cache --no-progress-bar --lpt 763.90s user 17.19s system 1109% cpu 1:10.41 totalOn Symfony framework: Before : 4 minutes ➜ time bin/rector --no-diffs --dry-run --clear-cache --no-progress-bar ../symfony/src
[OK] 2683 files would have been changed (dry-run) by Rector
bin/rector --no-diffs --dry-run --clear-cache --no-progress-bar ../symfony/sr 2348.09s user 118.16s system 1026% cpu 4:00.18 totalAfter (with ➜ time bin/rector --no-diffs --dry-run --clear-cache --no-progress-bar ../symfony/src --lpt
[OK] 2683 files would have been changed (dry-run) by Rector
bin/rector --no-diffs --dry-run --clear-cache --no-progress-bar ../symfony/sr 1327.72s user 163.84s system 1035% cpu 2:23.99 totalThe summary:For laravel: this PR |
e652152 to
ad7c011
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The experimental path has unresolved result-loss and error-limit defects alongside the acknowledged memory trade-off.
4 open findings
What changed in this PR
Proof of concept for an opt-in LPT scheduler that balances files by size, supports work stealing, and retains warm workers for improved parallel performance.
Changes:
- Adds LPT bucket scheduling and experimental processing behind
--lpt. - Extracts shared worker-result collection logic.
- Documents benchmarks, memory costs, and unresolved risks.
| File | Description |
|---|---|
src/Parallel/Experimental/ValueObject/BucketSchedule.php |
Stores per-worker job buckets. |
src/Parallel/Experimental/LptScheduleFactory.php |
Creates size-balanced LPT buckets. |
src/Parallel/Experimental/ExperimentalParallelFileProcessor.php |
Runs persistent workers with work stealing. |
src/Parallel/Application/ParallelResultCollector.php |
Centralizes result aggregation. |
src/Parallel/Application/ParallelFileProcessor.php |
Uses the shared result collector. |
src/Console/ProcessConfigureDecorator.php |
Registers --lpt. |
src/Configuration/Option.php |
Defines the LPT option. |
src/Application/ApplicationFileProcessor.php |
Selects the experimental scheduler. |
PARALLEL_LPT_POC_NOTES.md |
Records design, benchmarks, and risks. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| ))); | ||
| } | ||
|
|
||
| return $parallelResultCollector->createProcessResult(); |
| &$bucketKeyByIdentifier, | ||
| $postFileCallback, | ||
| &$systemErrorsCount, | ||
| &$reachedInternalErrorsCountLimit, |
| @@ -0,0 +1,349 @@ | |||
| # PoC: LPT bucket scheduling for parallel run | |||
| /** | ||
| * @experimental Opt-in to the LPT bucket scheduler | ||
| * @see \Rector\Parallel\Experimental\LptScheduleFactory | ||
| * @var string |
Created with the help of Claude.
A proof of concept for testing, not a merge candidate. No tests yet, an opt-in experimental flag, and one unresolved trade-off (memory) that would have to be settled before this could ship. Opening it to put the approach and the numbers in front of people who can try it on their own codebases.
Follows up on #8494, and specifically @samsonasik's comment there: #8494 (comment)
All of it behind
--lpt, opt-in. Without the flag the default path is untouched.jobSizesweep, LPT + stealing, 14 workers:Best current 152.2 s → best
--lpt90.7 s, i.e. -40 %. At identical worker count andjobSize, -25 %.Every run produces the same 3 338 diffs (sha over the file-sorted diff list) and the same 2 errors, except
jobSize: 150.